Image encoder and camera system

ABSTRACT

A compression controller is capable of executing prediction mode selection for selecting any one of forward prediction, backward prediction, or bidirectional prediction as a prediction mode used for encoding based on spatial frequencies of an image to be encoded. The compression controller determines whether or not to execute the prediction mode selection based on image mobility information indicating an amount of motion in the image to be encoded.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation of PCT International ApplicationPCT/JP2011/001332 filed on Mar. 7, 2011, which claims priority toJapanese Patent Application No. 2010-061521 filed on Mar. 17, 2010. Thedisclosures of these applications including the specifications, thedrawings, and the claims are hereby incorporated by reference in theirentirety.

BACKGROUND

The present disclosure relates to image encoders encoding moving images,and camera systems including the image encoders.

Conventionally, image encoders are known, which perform intra-frameencoding and inter-frame encoding using a system according to movingpicture experts group (MPEG) standards. In recent years, such imageencoders have the problem that the power consumption and the processingtime required for encoding increase with an increase in an image sizefor higher image quality. In order to solve the problem, InternationalPatent Publication No. WO 2005/076629 shows an image encoder whichselects one of a reference mode using bidirectional encoding, or areference mode not using bidirectional encoding, in accordance withencoding environment such as settings of resolution, a frame rate, etc.As a result, encoding according to the environment is executed.

SUMMARY

In International Patent Publication No. WO 2005/076629, however, imagequality is significantly degraded depending on a captured image, sincecharacteristics of the image is not reflected by the selection of thereference mode.

The present disclosure was made to solve the problem. It is an objectiveof the present disclosure to reduce deterioration in image qualitycaused by encoding of a moving image, and to reduce power consumptionand processing time required for the encoding.

In order to achieve the objective, an aspect of the present disclosureprovides an image encoder encoding a moving image. The encoder includesa compression processor configured to encode brightness data andcolor-difference data of an image to be encoded, and to output theencoded data; and a compression controller capable of executingprediction mode selection for selecting any one of forward prediction,backward prediction, or bidirectional prediction as a prediction modeused for the encoding based on spatial frequencies of the image to beencoded, and configured to perform scene determination for determiningwhether or not to execute the prediction mode selection based on imagemobility information indicating an amount of motion in the image to beencoded.

According to this aspect, the prediction mode used for the encoding isdetermined based on the spatial frequencies of the image to be encoded.Therefore, the prediction mode used for the encoding is properlydetermined to obtain the advantage of reducing deterioration in imagequality, power consumption, and processing time required for theencoding.

The present disclosure reduces deterioration in the image quality causedby encoding of a moving image, and reduces the power consumption and theprocessing time required for the encoding.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of a camerasystem according to a first embodiment of the present disclosure.

FIG. 2 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to the firstembodiment.

FIG. 3 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to a first variation ofthe first embodiment.

FIG. 4 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to a second variationof the first embodiment.

FIG. 5 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to a third variation ofthe first embodiment.

FIG. 6 is a block diagram illustrating the configuration of a camerasystem according to a second embodiment.

FIG. 7 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to the secondembodiment.

FIG. 8 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to a first variation ofthe second embodiment.

FIG. 9 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to a second variationof the second embodiment.

FIG. 10 is a flow chart illustrating the operation of a compressionprocessor and a compression controller according to a third embodiment.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described hereinafter withreference to the drawings. In the following embodiments, the samereference characters as those shown in the other embodiments are used torepresent equivalent operation, and the explanation thereof will beomitted.

First Embodiment

A camera system 100 according to a first embodiment of the presentdisclosure includes, as shown in FIG. 1, an imager 101, an imagingcontroller 102, an image processor 103, a memory 104, and a display 105.The camera system 100 is, for example, mounted in a network camera, avehicle camera, a digital video camera, and a digital still camera.

The imager 101 converts image light to an electrical signal using animaging element.

The imaging controller 102 takes in the electrical signal obtained bythe imaging element of the imager 101, performs analog-to-digitalconversion of the electrical signal, and outputs a video signal. Theimaging controller 102 controls timing of capturing the electricalsignal, and time for irradiating the imaging element of the imager 101with image light.

The image processor 103 includes an image input 106 and an image encoder107.

The image input 106 performs automatic exposure (AE), white balancing(WB), aperture processing, brightness color-difference signal (YC)processing, and denoising on the video signal output from the imagingcontroller 102, thereby generating brightness data and color-differencedata of an image. The image input 106 includes a motion detector 108 anda frequency detector 109. The motion detector 108 obtains the amount ofmotion (i.e., image mobility information) using representative matchingbased on the video signal output from the imaging controller 102. Thefrequency detector 109 detects spatial frequencies of the image shown bythe video signal from a plurality of frequency detection ranges based onthe video signal output from the imaging controller 102. The frequencydetection ranges are a plurality of divided regions of an image. Eachfrequency detection range (i.e., divided region) includes a plurality ofmacro blocks.

The image encoder 107 encodes the brightness data and thecolor-difference data generated by the image input 106, and outputs theencoded data. Specifically, the image encoder 107 includes a compressionprocessor 110, a compression controller 111, and a display interface(IF) 112.

The compression processor 110 encodes the brightness data and thecolor-difference data of the image to be encoded, which are generated bythe image input 106, and outputs the encoded data. The encoding isperformed in a mode according to the MPEG standards. In the encoding,the compression controller 111 controls which of reference images is tobe referenced. A motion vector and a code amount are obtained by theencoding.

The compression controller 111 selects the reference image referenced bythe compression processor 110 in the encoding based on the amount ofmotion obtained by the motion detector 108 and the spatial frequenciesobtained by the frequency detector 109. Then, the compression controller111 controls the compression processor 110 to perform the encoding withreference to the selected reference image. The compression controller111 includes a scene determiner 113 and an image difference determiner114.

The image difference determiner 114 selects the reference image based onthe spatial frequencies of the image to be encoded. That is, the imagedifference determiner 114 executes prediction mode selection forselecting a prediction mode used for the encoding of the compressionprocessor 110 from forward prediction, backward prediction, andbidirectional prediction based on the spatial frequencies of the imageto be encoded.

The scene determiner 113 determines whether or not the prediction modeselection by the image difference determiner 114 is performed on animage compressed as an image encoded in bidirectional prediction (i.e.,a picture with reference to a previous frame and a subsequent frame).The determination is made based on the amount of motion obtained by themotion detector 108.

The display IF 112 outputs the brightness data and the color-differencedata generated by the image input 106.

The memory 104 temporarily stores the encoded data output from the imageencoder 107 of the image processor 103.

The display 105 displays the image based on the brightness data and thecolor-difference data output from the display IF 112 of the imageencoder 107.

Next, detailed operation of the compression processor 110 and thecompression controller 111 will be described step by step with referenceto FIG. 2. The operation is performed on an image compressed as an imageencoded in bidirectional prediction (a picture with reference to aprevious and subsequent frame). The operation in FIG. 2 is performed oneach frame. That is, the reference image is selected on each frame. Thecontrol of the reference image is not changed from the start to the endof encoding for one frame. In step 201, the scene determiner 113determines whether or not the amount of motion obtained by the motiondetector 108 is equal to or larger than a predetermined value. Where theamount of motion is equal to or larger than the predetermined value, thescene determiner 113 determines not to allow the image differencedeterminer 114 to perform prediction mode selection. Where the amount ofmotion is equal to or larger than the predetermined value, the processproceeds to step 204. Where the amount of motion is smaller than thepredetermined value, the process proceeds to step 202.

In step 202, the image difference determiner 114 reads the spatialfrequencies obtained by the frequency detector 109, and determineswhether or not the average of the spatial frequencies in the frequencydetection ranges is equal to or greater than a predetermined value.Where the average is equal to or greater than the predetermined value,the process proceeds to step 205. Where the average is less than thepredetermined value, the process proceeds to step 203.

In step 203, the image difference determiner 114 selects one of theprevious image or the subsequent image of an image to be encoded in thisembodiment, of which the average of the spatial frequencies in thefrequency detection ranges is closer to that of the image to be encoded.Where the previous image is selected, the process proceeds to step 206.Where the subsequent image is selected, the process proceeds to step207.

In step 204, the compression controller 111 controls the compressionprocessor 110 to perform encoding using all the prediction modes of theforward prediction, the backward prediction, and the bidirectionalprediction. In accordance with this control, the compression processor110 performs the encoding. At this time, the previous image and thesubsequent image need to be input to the compression processor 110.

In step 205, the compression controller 111 controls the compressionprocessor 110 to perform encoding using both the previous image and thesubsequent image as the reference images. That is, the compressioncontroller 111 controls the compression processor 110 to perform theencoding using the bidirectional prediction. In accordance with thiscontrol, the compression processor 110 performs the encoding, anddetermines the most efficient prediction mode (i.e., the prediction modewhich provides the least information amount of data by the encoding).The compression processor 110 outputs data obtained by the encodingusing the prediction mode as encoded data. At this time, the previousimage and the subsequent image need to be input to the compressionprocessor 110. For decoding the image encoded in this manner, only oneof the reference images is needed, which is selected based on thedifference between the image to be decoded and the reference imagedetermined by the compression controller 111. This reduces the number ofmemory access in the decoding.

In step 206, the compression controller 111 controls the compressionprocessor 110 to perform encoding using only the previous image as thereference image. That is, the compression controller 111 controls thecompression processor 110 to perform the encoding using the forwardprediction. In accordance with this control, the compression processor110 performs the encoding. At this time, the previous image needs to beinput to the compression processor 110. For decoding the image encodedin this manner, only the previous image is needed. This reduces thenumber of memory access in both the encoding and the decoding.

In step 207, the compression controller 111 controls the compressionprocessor 110 to perform encoding using only the subsequent image as thereference image. That is, the compression controller 111 controls thecompression processor 110 to perform the encoding using the backwardprediction. In accordance with this control, the compression processor110 performs the encoding. At this time, the subsequent image needs tobe input to the compression processor 110. For decoding the imageencoded in this manner, only the subsequent image is needed. Thisreduces the number of memory access in both the encoding and thedecoding.

In this embodiment, where the amount of motion in the image is small andthe spatial frequencies are low, one of the reference images isreferenced in encoding. This reduces the number of memory access in theencoding and decoding, thereby reducing power consumption. In addition,where the amount of motion in the image is small and the spatialfrequencies are low, image quality is less influenced by the reductionin the reference images, thereby reducing deterioration in the imagequality.

First Variation of First Embodiment

In a camera system 100 according to a first variation of the firstembodiment, a compression processor 110 and a compression controller 111execute operation shown in FIG. 3 in place of the operation shown inFIG. 2. As shown in the figure, in this variation, the compressioncontroller 111 executes operation of step 301 in place of step 203.

In step 301, an image difference determiner 114 selects one of theprevious image or the subsequent image of an image to be encoded in thisvariation, of which the integrated value of spatial frequencies in aplurality of frequency detection ranges is closer to that of the imageto be encoded. Where the previous image is selected, the processproceeds to step 206. Where the subsequent image is selected, theprocess proceeds to step 207.

Since the other configurations and operation are the same as those inthe first embodiment, the same reference characters are used torepresent equivalent operation in FIG. 3, and the explanation thereof isomitted.

Second Variation of First Embodiment

In a camera system 100 according to a second variation of the firstembodiment, a compression processor 110 and a compression controller 111execute operation shown in FIG. 4 in place of the operation shown inFIG. 2. As shown in the figure, in this variation, the compressioncontroller 111 executes operation of step 401 in place of step 203.

In step 401, an image difference determiner 114 selects one of theprevious image or the subsequent image of an image to be encoded in thisvariation, which has a smaller maximum difference in spatial frequenciesin a plurality of frequency detection ranges from that of the image tobe encoded. Where the previous image is selected, the process proceedsto step 206. Where the subsequent image is selected, the processproceeds to step 207.

Since the other configurations and operation are the same as those inthe first embodiment, the same reference characters are used torepresent equivalent operation in FIG. 4, and the explanation thereof isomitted.

Third Variation of First Embodiment

In a camera system 100 according to a third variation of the firstembodiment, a compression processor 110 and a compression controller 111execute operation shown in FIG. 5 in place of the operation shown inFIG. 2. As shown in the figure, in this variation, the compressioncontroller 111 executes operation of step 501 in place of step 203.

In step 501, the image difference determiner 114 selects the one of theprevious image or the subsequent image of an image to be encoded in thisvariation, which has more frequency detection ranges, in which spatialfrequencies having greater differences from spatial frequencies of theimage to be encoded than a predetermined threshold are detected. Wherethe previous image is selected, the process proceeds to step 206. Wherethe subsequent image is selected, the process proceeds to step 207.

Since the other configurations and operation are the same as those inthe first embodiment, the same reference characters are used torepresent equivalent operation in FIG. 5, and the explanation thereof isomitted.

Second Embodiment

In a camera system 100 according to a second embodiment of the presentdisclosure, as shown in FIG. 6, a compression controller 111 includes acode amount accumulator 601 in addition to a scene determiner 113 and animage difference determiner 114. The code amount accumulator 601accumulates a code amount obtained by encoding of a compressionprocessor 110.

In the camera system 100 according to the second embodiment, thecompression processor 110 and the compression controller 111 executeoperation shown in FIG. 7 in place of the operation shown in FIG. 2. Asshown in the figure, in the second embodiment, the compressioncontroller 111 executes operation of step 701 before step 201.

In step 701, the scene determiner 113 determines whether or not the codeamount accumulated in the code amount accumulator 601 is larger than apredetermined threshold. Where the code amount is larger than thepredetermined threshold, the scene determiner 113 determines that theimage is in a high compression state. Where the code amount is notlarger than the predetermined threshold, the scene determiner 113determines that the image is in a normal compression state. Where theimage is determined as being in the high compression state, the processproceeds to step 204. Where the image is determined as being in thenormal compression state, the process proceeds to step 201.

The threshold used in the determination of step 701 may be a fixedvalue, or may be controlled based on the bit rate of a target system.The threshold may be controlled to be high when the bit rate is high,and low when the bit rate is low, thereby efficiently selecting areference image.

Since the other configurations and operation are the same as those inthe first embodiment, the same reference characters are used torepresent equivalent configurations and operation in FIGS. 6 and 7, andthe explanation thereof is omitted.

According to the second embodiment, deterioration in image qualitycaused in a high compression state can be controlled, and the number ofmemory access in encoding and decoding can be reduced, thereby reducingpower consumption.

First Variation of Second Embodiment

In a camera system 100 according to a first variation of the secondembodiment, a compression processor 110 and a compression controller 111execute operation shown in FIG. 8 in place of the operation shown inFIG.7. As shown in the figure, in this variation, the compressioncontroller 111 executes operation of step 801 in place of step 701.

In step 801, a scene determiner 113 determines whether or not acompression ratio used in encoding by the compression processor 110 islower than a predetermined threshold. Where the ratio is lower than thethreshold, the scene determiner 113 determines that an image is in ahigh compression state. Where the ratio is not lower than the threshold,the scene determiner 113 determines that the image is in a normalcompression state. Where the image is determined as being in the highcompression state, the process proceeds to step 204. Where the image isdetermined as being in the normal compression state, the processproceeds to step 201.

Similar to the second embodiment, the threshold used in thedetermination of step 801 may be a fixed value, or may be controlledbased on the bit rate of a target system.

Since the other configurations and operation are the same as those inthe second embodiment, the same reference characters are used torepresent equivalent operation in FIG. 8, and the explanation thereof isomitted.

Second Variation of Second Embodiment

In a camera system 100 according to a second variation of the secondembodiment, a compression processor 110 and a compression controller 111execute operation shown in FIG. 9 in place of the operation shown inFIG.7. As shown in the figure, in this variation, the compressioncontroller 111 executes operation of step 901 in place of step 701.

In step 901, a scene determiner 113 determines whether or not a bit rateof encoding by the compression processor 110 is higher than apredetermined threshold. Where the rate is higher than the threshold,the scene determiner 113 determines that an image is in a highcompression state. Where the rate is not higher than the threshold, thescene determiner 113 determines that the image is in a normalcompression state. Where the image is determined as being in the highcompression state, the process proceeds to step 204. Where the image isdetermined as being in the normal compression state, the processproceeds to step 201.

Since the other configurations and operation are the same as those inthe second embodiment, the same reference characters are used torepresent equivalent operation in FIG. 9, and the explanation thereof isomitted.

Third Embodiment

In a camera system 100 according to a third embodiment, a compressionprocessor 110 and a compression controller 111 execute operation shownin FIG. 10 in place of the operation shown in FIG.7. As shown in thefigure, in the third embodiment, in step 701, where a code amountaccumulated in a code amount accumulator 601 is determined as being notlarger than a predetermined threshold, the process proceeds to step1001.

In step 1001, the compression controller 111 determines whether or notthe code amount accumulated in the code amount accumulator 601 issmaller than a predetermined threshold. Where the code amount is smallerthan the predetermined threshold, the compression controller 111determines that the image is in a low compression state. Where the codeamount is not smaller than the predetermined threshold, the compressioncontroller 111 determines that the image is in a normal compressionstate. Where the image is determined as being in the low compressionstate, the process proceeds to step 203. Where the image is determinedas being in the normal compression state, the process proceeds to step201.

Since the other configurations and operation are the same as those inthe second embodiment, the same reference characters are used torepresent equivalent operation in FIG. 10, and the explanation thereofis omitted.

According to the third embodiment, one of the previous image or thesubsequent image is selected as a reference image when the image is in alow compression state in which the amount of the decoded data lessinfluences deterioration in image quality. This efficiently reduces thenumber of memory access in encoding and decoding. Therefore,deterioration in the image quality can be controlled, and powerconsumption can be efficiently controlled.

In the first to third embodiments and their variations, the selection ofthe reference image by the compression controller 111 (i.e., thedetermination by the scene determiner 113) is performed on each frame.However, it may be performed in each divided region such as a macroblock, forming part of a frame. This enables highly precise selection ofthe reference image.

In the first to third embodiments and their variations, in step 201,whether or not the prediction mode selection is performed based on theamount of motion obtained by the motion detector 108. The determinationmay be made based on angular velocity information obtained by a lenscontroller (not shown) or the motion vector of the image, which is animage immediately before than the image to be encoded and is obtained bythe compression processor 110. The determination may be made based ontwo or all of the amount of motion obtained by the motion detector 108,the angular velocity information, and the motion vector. Where themotion vector obtained by the compression processor 110 is used for thedetermination on whether or not to perform prediction mode selection,determination in each macro block is facilitated.

The information (e.g., an angular velocity of a camera, a motion vector(i.e., image motion information), spatial frequencies, etc.) used in aconventional camera may be used for the prediction mode selection, orthe determination on whether or not to perform the prediction modeselection, thereby mitigating an increase in the number of processing.

The compression information (e.g., the code amount, the bit rate, etc.)obtained in the encoding may be used for the prediction mode selection,or the determination on whether or not to perform the prediction modeselection, thereby selecting the most preferable prediction modeaccording to rate control in a normal mode of taking a moving image.

In the first to third embodiments and their variations, in step 202,which of step 205 and step 203 the process proceeds to is determinedbased on whether or not the average of the spatial frequencies in thefrequency detection ranges is equal to or greater than the predeterminedvalue. However, the determination may be made based on whether thenumber of the spatial frequencies higher than a predetermined thresholdis equal to or larger than a predetermined number in the frequencydetection ranges. Where the selection of the reference image by thecompression controller 111 (the determination by the scene determiner113) is performed in each frequency detection range (i.e., dividedregion), the determination may be made based on whether or not thespatial frequency in the frequency detection range is equal to or higherthan a predetermined value.

The present disclosure is applicable to encoding systems other than asystem according to the MPEG standards, which use bidirectionalprediction.

The present disclosure is useful as an image encoder encoding a movingimage, and a camera system including the image encoder.

1. An image encoder encoding a moving image, the encoder comprising: acompression processor configured to encode brightness data andcolor-difference data of an image to be encoded, and to output theencoded data; and a compression controller capable of executingprediction mode selection for selecting any one of forward prediction,backward prediction, or bidirectional prediction as a prediction modeused for the encoding based on spatial frequencies of the image to beencoded, and configured to perform scene determination for determiningwhether or not to execute the prediction mode selection based on imagemobility information indicating an amount of motion in the image to beencoded.
 2. The image encoder of claim 1, wherein the image mobilityinformation includes pixel mobility information obtained byrepresentative matching.
 3. The image encoder of claim 1, wherein theimage encoder is mounted in a camera, and the image mobility informationincludes angular velocity information of the camera.
 4. The imageencoder of claim 1, wherein the image mobility information includes amotion vector obtained for an image which is an image immediately beforethan the image to be encoded.
 5. The image encoder of claim 1, whereinthe image mobility information includes at least two of pixel motioninformation obtained by representative matching, angular velocityinformation, and a motion vector obtained for an image which is an imageimmediately before than the image to be encoded.
 6. The image encoder ofclaim 1, wherein in the prediction mode selection, when either one offorward prediction or backward prediction is selected as a mode of theencoding, an average of spatial frequencies in a plurality of frequencydetection ranges of a reference image of the selected one is closer tothat of the image to be encoded.
 7. The image encoder of claim 1,wherein in the prediction mode selection, when either one of forwardprediction or backward prediction is selected as a mode of the encoding,an integrated value of spatial frequencies in a plurality of frequencydetection ranges of a reference image of the selected one is closer tothat of the image to be encoded.
 8. The image encoder of claim 1,wherein in the prediction mode selection, when either one of forwardprediction or backward prediction is selected as a mode of the encoding,the selected one has a smaller maximum difference in spatial frequenciesin a plurality of frequency detection ranges between a reference imageand the image to be encoded.
 9. The image encoder of claim 1, wherein inthe prediction mode selection, when either one of forward prediction orbackward prediction is selected as a mode of the encoding, the selectedone has more frequency detection ranges of a reference image, in whichspatial frequencies having greater differences from the spatialfrequencies of the image to be encoded than a threshold are detected.10. The image encoder of claim 1, wherein in the scene determination,whether or not to execute the prediction mode selection is determinedbased on a compression state criterion value indicating a code amount ofthe image, in addition to the image mobility information.
 11. The imageencoder of claim 1, wherein in the scene determination, whether or notto execute the prediction mode selection is determined based on acompression state criterion value indicating a compression ratio in theencoding, in addition to the image mobility information.
 12. The imageencoder of claim 1, wherein in the scene determination, whether or notto execute the prediction mode selection is determined based on acompression state criterion value indicating a bit rate of the encoding,in addition to the image mobility information.
 13. The image encoder ofclaim 10, wherein in the scene determination, whether or not thecompression state criterion value is greater than a first threshold isdetermined in first determination, and whether or not to execute theprediction mode selection is determined based on a result of the firstdetermination.
 14. The image encoder of claim 13, wherein in the scenedetermination, whether or not the compression state criterion value issmaller than a second threshold is determined in second determination,and whether or not to execute the prediction mode selection isdetermined based a result of the second determination.
 15. The imageencoder of claim 1, wherein the scene determination is made on eachframe by the compression controller.
 16. The image encoder of claim 1,wherein the scene determination is made in each divided region formingpart of a frame by the compression controller.
 17. A camera systemcomprising: the image encoder of claim 1; an imager configured toconvert image light to an electrical signal using an imaging element; animaging controller configured to control timing of capturing theelectrical signal obtained by the imaging element and a time forirradiating the imaging element with image light, to performanalog-to-digital conversion of the electrical signal, and to output theconverted signal as an image signal; an image input configured togenerate brightness data and color-difference data based on the imagesignal output from the imaging controller; a memory configured totemporarily store the encoded data output from the compression processorof the image encoder; and a display configured to display an image basedon the brightness data and the color-difference data generated by theimage input of the image encoder.